Systems and methods for device-aware flexible tool registration

ABSTRACT

A medical system comprises a flexible device configured to be positioned at least partially within an anatomical passageway of a plurality of anatomical passageways within a patient anatomy. The medical system further comprises a memory device including computer executable instructions. The computer executable instructions are for performing operations comprising determining a deformation force exerted by a section of the flexible device. The operations further comprise registering a model of a candidate anatomical passageway of the plurality of anatomical passageways to a model of the flexible device based on: a shape of the flexible device; and the deformation force exerted by the section of the flexible device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/815,328, filed Mar. 11, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/101,518, filed Jun. 3, 2016, now U.S. Pat. No.10,610,306, which is the U.S. National Phase of International PatentApplication No. PCT/US2014/069228, filed Dec. 9, 2014, which designatedthe U.S. and claims priority to and the benefit of U.S. ProvisionalPatent Application No. 61/913,747, filed Dec. 9, 2013, entitled “Systemsand Methods for Device-Aware Flexible Tool Registration,” all of whichare incorporated by reference herein in their entireties.

FIELD

The present disclosure is directed to systems and methods for tracking amedical device within a patient anatomy during a medical procedure, andmore particularly to systems and methods for efficiently tracking amedical device within a patient anatomy using a shape sensor and/or aposition sensor.

BACKGROUND

Minimally invasive medical techniques are intended to reduce the amountof tissue that is damaged during medical procedures, thereby reducingpatient recovery time, discomfort, and harmful side effects. Suchminimally invasive techniques may be performed through natural orificesin a patient anatomy or through one or more surgical incisions.Clinicians may insert medical tool through these natural orifices orincisions to reach a target tissue location. Medical tools includeinstruments such as therapeutic instruments, diagnostic instruments, andsurgical instruments. To reach the target tissue location, a minimallyinvasive medical tool may navigate natural or surgically createdpassageways in anatomical systems such as the lungs, the colon, theintestines, the kidneys, the heart, the circulatory system, or the like.

Minimally invasive surgical procedures typically rely on some sort ofinstrument position monitoring to ensure proper access to, and behaviorat, the target tissue location. Conventional minimally invasive surgicalinstruments are generally either formed from generally rigid, elongateelements (e.g., laparoscopic or teleoperational systems) or highlyflexible systems designed to follow a predetermined anatomic path (e.g.,angioplasty balloon catheters). in either case, position monitoringtypically involves localized tracking of a discrete portion of theinstrument (e.g., the distal tip of a catheter). The remainingguidewire/catheter length is not actively monitored, except in anincidental sense to the extent the remaining length is shown duringfluoroscopic visualization of the tip advancement.

However, increasingly more complex minimally invasive surgical systemscan require enhanced instrument position monitoring for safe andeffective use. Navigational assist systems help the clinician route thesurgical instruments and avoid damage to the anatomy. These systems canincorporate the use of shape sensors to more accurately describe theshape, pose, and location of the surgical instrument in real space orwith respect to pre-procedural or concurrent images. In a dynamicanatomical system and/or in an anatomical region dense with manyanatomical passageways, accurately registering the minimally invasiveinstrument to the anatomical system is a time consuming and processingintensive task. Improved systems and methods are needed for increasingthe accuracy and efficiency of systems and methods of registeringminimally invasive instruments to the anatomical system.

SUMMARY

In one aspect, the present disclosure describes a method of determiningthe accurate positioning of a device within a patient anatomy. Themethod comprises generating a first model of at least one anatomicalpassageway from anatomical data describing the patient anatomy. In oneaspect, the method comprises determining a shape of a device positionedwithin branched anatomical passageways of the patient anatomy, thedevice including a plurality of sections, each section of the pluralityof sections having a distinct physical property, and computing a set ofdeformation forces for each section of the plurality of sections of thedevice. In one aspect, the method comprises generating a second model ofthe device positioned within the plurality of branched anatomicalpassageways by adjusting the first model based upon the determined shapeof the device and the set of deformation forces for each section of theplurality of sections of the device.

In another aspect. the present disclosure describes a medical systemcomprising a flexible device including a shape sensor, a memory, and anon-transitory computer readable media. In one aspect, the memory canstore anatomical data describing a patient anatomy, and the non-transitory computer readable media can contain computer executableinstructions for registering the flexible device to the anatomical datadescribing the patient anatomy. In one aspect, the computer executableinstructions include instructions for generating a first model of aplurality of branched anatomical passageways from the stored anatomicaldata describing the patient anatomy, instructions for receivinginformation from the shape sensor to determine a shape of the devicepositioned within the plurality of branched anatomical passageways, thedevice including a plurality of sections, each section of the pluralityof sections having a distinct physical property, instructions forcomputing a set of deformation forces for each section of the pluralityof different sections of the device, and/or instructions for generatinga second model of the device positioned within the plurality of branchedanatomical passageways by adjusting the first model based upon thedetermined shape of the device and the set of deformation forces foreach section of the plurality of sections of the device.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Aspects of the present disclosure arc best understood from the followingdetailed description when read with the accompanying figures. It isemphasized that, in accordance with the standard practice in theindustry, various features are not drawn to scale. In fact, thedimensions of the various features may be arbitrarily increased orreduced for clarity of discussion. In addition, the present disclosuremay repeat reference numerals and/or letters in the various examples.This repetition is for the purpose of simplicity and clarity and doesnot in itself dictate a relationship between the various embodimentsand/or configurations discussed.

FIG. 1 is a teleoperational medical system, in accordance withembodiments of the present disclosure.

FIG. 2 illustrates a medical instrument system utilizing aspects of thepresent disclosure.

FIG. 3A is an image of a human lung registered with an image of aminimally invasive instrument apparently positioned outside ananatomical passageway.

FIG. 3B is an image of a human lung registered with an image of aminimally invasive instrument apparently positioned inside an incorrectanatomical passageway.

FIG. 3C is an internal image of the human lung depicting the region ofthe human lung from the viewpoint of the minimally invasive instrument.

FIG. 4A is an illustration of model of a bronchial passageway of a lungwith a catheter.

FIG. 4B is an illustration of the model of FIG. 4A adjusted based upondeformation forces applied by the catheter.

FIG. 5A illustrates a composite image of an undeformed model ofbronchial passageways and a sensed model of a catheter.

FIGS. 5B-5D illustrate composite images of various deformed models ofthe bronchial passageways showing the advancement of the catheter intodifferent passageways.

FIG. 6 is a diagram of a sensed model of the catheter shown in FIGS.5A-5D in comparison with two different models representing two possiblepassageways that could contain the catheter.

FIG. 7A illustrates a composite image of an undeformed model ofbronchial passageways and a sensed model of a catheter.

FIGS. 7B-7D illustrate composite images of the model of the bronchialpassageways adjusted with the advancement of the catheter.

FIG. 8 illustrates a composite image of a deformed model of thebronchial passageway and the catheter shown in FIGS. 7A-7D.

FIG. 9 is a flowchart illustrating a method for deforming a model ofanatomic passageways based upon the shape of the catheter, thedeformative forces applied by the different sections of the catheter,and the restorative forces of the passageways.

FIG. 10 is an illustration of a model of a bronchial passageway of alung in states of expiration and inspiration.

FIG. 11 is a flowchart illustrating a method of deforming a model of thebronchial passageway for states of inspiration and expiration.

FIG. 12 includes illustrations of models of an anatomical passagewayaccording to another embodiment of the disclosure.

FIG. 13 is a histogram for matching the model of FIG. 12 .

FIG. 14 illustrates point sets used to determine a transformation.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of thepresent disclosure, reference will now be made to the embodimentsillustrated in the drawings, and specific language will be used todescribe the same. It will nevertheless be understood that no limitationof the scope of the disclosure is intended. In the following detaileddescription of the aspects of the invention, numerous specific detailsare set forth in order to provide a thorough understanding of thedisclosed embodiments. However, it will be obvious to one skilled in theart that the embodiments of this disclosure may be practiced withoutthese specific details. In other instances well known methods,procedures, components, and circuits have not been described in detailso as not to unnecessarily obscure aspects of the embodiments of theinvention.

Any alterations and further modifications to the described devices,instruments, methods, and any further application of the principles ofthe present disclosure are fully contemplated as would normally occur toone skilled in the art to which the disclosure relates. In particular,it is fully contemplated that the features, components, and/or stepsdescribed with respect to one embodiment may be combined with thefeatures, components, and/or steps described with respect to otherembodiments of the present disclosure. In addition, dimensions providedherein arc for specific examples and it is contemplated that differentsizes, dimensions, and/or ratios may be utilized to implement theconcepts of the present disclosure. To avoid needless descriptiverepetition, one or more components or actions described in accordancewith one illustrative embodiment can be used or omitted as applicablefrom other illustrative embodiments. For the sake of brevity, thenumerous iterations of these combinations will not be describedseparately. For simplicity, in some instances the same reference numbersare used throughout the drawings to refer to the same or like parts.

The embodiments below will describe various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian X, Y, Z coordinates). As usedherein, the term “orientation” refers to the rotational placement of anobject or a portion of an object (three degrees of rotational freedome.g., roll, pitch, and yaw). As used herein, the term “pose” refers tothe position of an object or a portion of an object in at least onedegree of translational freedom and to the orientation of that object orportion of the object in at least one degree of rotational freedom (upto six total degrees of freedom). As used herein, the term “shape”refers to a set of poses, positions, or orientations measured along anobject.

Referring to FIG. 1 of the drawings, a teleoperational medical systemfor use in, for example, medical procedures including diagnostic,therapeutic, or surgical procedures, is generally indicated by thereference numeral 100. As will be described, the teleoperational medicalsystems of this disclosure are under the teleoperational control of asurgeon. In alternative embodiments. a teleoperational medical systemmay be under the partial control of a computer programmed to perform theprocedure or sub-procedure. In still other alternative embodiments, afully automated medical system, under the full control of a computerprogrammed to perform the procedure or sub-procedure, may be used toperform procedures or sub-procedures.

As shown in FIG. 1 , the teleoperational system 100 generally includes ateleoperational assembly 102 for operating a medical instrument system104 in performing various procedures on the patient P. The assembly 102is mounted to or near an operating table O on which a patient P ispositioned. The medical instrument system 104 is operably coupled to theteleoperational assembly 102. An operator input system 106 allows asurgeon or other type of clinician S to view images of or representingthe surgical site and to control the operation of the medical instrumentsystem 104.

In alternative embodiments, the teleoperational system may include morethan one manipulator assembly. The exact number of manipulatorassemblies will depend on the surgical procedure and the spaceconstraints within the operating room among other factors. The operatorinput system 106 may be located at a surgeon's console C, which isusually located in the same room as operating table O. However, itshould be understood that the surgeon S can be located in a differentroom or a completely different building from the patient P. Operatorinput system 106 generally includes one or more control device(s) 112for controlling the medical instrument system 104. The control device(s)112 may include one or more of any number of a variety of input devices,such as hand grips, joysticks, trackballs, data gloves, trigger-guns,hand-operated controllers, voice recognition devices, touch screens,body motion or presence sensors, and the like. In some embodiments, thecontrol device(s) 112 will be provided with the same degrees of freedomas the medical instruments of the teleoperational assembly to providethe surgeon with telepresence, the perception that the control device(s)are integral with the instruments so that the surgeon has a strong senseof directly controlling instruments as if present at the surgical site.In other embodiments, the control device(s) 112 may have more or fewerdegrees of freedom than the associated medical instruments and stillprovide the surgeon with telepresence. In some embodiments, the controldevice(s) 112 are manual input devices which move with six degrees offreedom, and which may also include an actuatable handle for actuatinginstruments (for example, for closing grasping jaws, applying anelectrical potential to an electrode, delivering a medicinal treatment,and the like).

The teleoperational assembly 102 supports the medical instrument system104 and may include a kinematic structure of one or more non-servocontrolled links (e.g., one or more links that may be manuallypositioned and locked in place, generally referred to as a set-upstructure) and a teleoperational manipulator. The teleoperationalassembly 102 includes plurality of actuators or motors that drive inputson the medical instrument system 104 in response to commands from thecontrol system (e.g., a control system 116). The motors include drivesystems that when coupled to the medical instrument system 104 mayadvance the medical instrument into a naturally or surgically createdanatomical orifice. Other motorized drive systems may move the distalend of the medical instrument in multiple degrees of freedom, which mayinclude three degrees of linear motion (e.g., linear motion along the X,Y, Z Cartesian axes) and in three degrees of rotational motion (e.g.,rotation about the X, Y, Z Cartesian axes). Additionally, the motors canbe used to actuate an articulable end effector of the instrument forgrasping tissue in the jaws of a biopsy device or the like.

The teleoperational medical system 100 also includes a sensor system 108with one or more sub-systems for receiving information about theinstruments of the teleoperational assembly. Such sub-systems mayinclude a position sensor system (e.g., an electromagnetic (EM) sensorsystem); a shape sensor system for determining the position,orientation, speed, velocity, pose, and/or shape of the catheter tipand/or of one or more segments along a flexible body of instrumentsystem 104; and/or a visualization system 110 for capturing images fromthe distal end of the catheter system.

The visualization system 110 may include a viewing scope assembly(described in greater detail below) such that a concurrent or real-timeimage of the surgical site is provided to surgeon console C. Theconcurrent image may be, for example, a two or three dimensional imagecaptured by an endoscope positioned within the surgical site. In thisembodiment, the visualization system 100 includes endoscopic componentsthat may be integrally or removably coupled to the medical instrument104. However in alternative embodiments, a separate endoscope, attachedto a separate manipulator assembly may be used with the surgicalinstrument to image the surgical site. The visualization system 110 maybe implemented as hardware, firmware, software or a combination thereofwhich interact with or are otherwise executed by one or more computerprocessors, which may include the processors of a control system 116(described below).

The teleoperational medical system 100 also includes a display system111 for displaying an image or representation of the surgical site andmedical instrument system(s) 104 generated by sub-systems of the sensorsystem 108. The display 111 and the operator input system 106 may beoriented so the operator can control the medical instrument system 104and the operator input system 106 with the perception of telepresence.

The display system 111 may also display an image of the surgical siteand surgical instruments captured by the visualization system 110. Thedisplay 111 and the control devices 112 may be oriented such that therelative positions of the imaging device in the scope assembly and thesurgical instruments are similar to the relative positions of thesurgeon's eyes and hands so the operator can manipulate the surgicalinstrument 104 and the hand control as if viewing the workspace insubstantially true presence. By true presence, it is meant that thepresentation of an image is a true perspective image simulating theviewpoint of an operator that is physically manipulating the instrument104.

Alternatively or additionally, the display 111 may present images of thesurgical site recorded and/or modeled preoperatively using imagingtechnology such as, computerized tomography (CT), magnetic resonanceimaging (MRI), fluoroscopy, thermography, ultrasound, optical coherencetomography (OCT), thermal imaging, impedance imaging, laser imaging, ornanotube X-ray imaging. The presented preoperative images may includetwo-dimensional, three-dimensional, or four-dimensional images. Thepresented preoperative or intra-operative images may includetwo-dimensional, three-dimensional, or four-dimensional (including e.g.,time based or velocity based information) images and associated imagedata sets for reproducing the images.

In some embodiments, the display 111 may display a virtual navigationalimage in which the actual location of the medical instrument 104 isregistered (i.e., dynamically referenced) with preoperative orconcurrent images to present the surgeon S with a virtual image of theinternal surgical site at the location of the tip of the instrument 104.An image of the tip of the instrument 104 or other graphical oralphanumeric indicators may be superimposed on the virtual image toassist the surgeon controlling the surgical instrument. Alternatively,the instrument 104 may not be visible in the virtual image.

In other embodiments, the display 111 may display a virtual navigationalimage in which the actual location of the surgical instrument isregistered with preoperative or concurrent images to present the surgeonS with a virtual image of surgical instrument within the surgical sitefrom an external viewpoint. An image of a portion of the surgicalinstrument or other graphical or alphanumeric indicators may besuperimposed on the virtual image to assist the surgeon controlling theinstrument 104.

The teleoperational medical system 100 also includes a control system112. The control system 112 includes at least one memory and at leastone processor (not shown), and typically a plurality of processors, foreffecting control between the medical instrument system 104, theoperator input system 106, the sensor system 108, and the display system110. The control system 112 also includes programmed instructions (e.g.,a computer-readable medium storing the instructions) to implement someor all of the methods described in accordance with aspects disclosedherein. While control system 112 is shown as a single block in thesimplified schematic of FIG. 1 , the system may include two or more dataprocessing circuits with one portion of the processing optionally beingperformed on or adjacent the teleoperational assembly 102, anotherportion of the processing being performed at the operator input system106, and the like. Any of a wide variety of centralized or distributeddata processing architectures may be employed. Similarly, the programmedinstructions may be implemented as a number of separate programs orsubroutines, or they may be integrated into a number of other aspects ofthe teleoperational systems described herein. In one embodiment, controlsystem 112 supports wireless communication protocols such as Bluetooth,IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

In some embodiments, control system 112 may include one or more servocontrollers that receive force and/or torque feedback from the medicalinstrument system 104. Responsive to the feedback, the servo controllerstransmit signals to the operator input system 106. The servocontroller(s) may also transmit signals instructing teleoperationalassembly 102 to move the medical instrument system(s) 104 which extendinto an internal surgical site within the patient body via openings inthe body. Any suitable conventional or specialized servo controller maybe used. A servo controller may be separate from, or integrated with,teleoperational assembly 102. In some embodiments, the servo controllerand teleoperational assembly are provided as part of a teleoperationalarm cart positioned adjacent to the patient's body.

The control system 112 may further include a virtual visualizationsystem to provide navigation assistance to the medical instrumentsystem(s) 104. Virtual navigation using the virtual visualization systemis based upon reference to an acquired dataset associated with the threedimensional structure of the anatomical passageways. More specifically,the virtual visualization system processes images of the surgical siteimaged using imaging technology such as computerized tomography (CT),magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound,optical coherence tomography (OCT), thermal imaging, impedance imaging,laser imaging, nanotube X-ray imaging, or the like. Software is used toconvert the recorded images into a two dimensional or three dimensionalcomposite representation of a partial or an entire anatomical organ oranatomical region. An image data set is associated with the compositerepresentation. The composite representation and the image data setdescribe the various locations and shapes of the passageways and theirconnectivity. The images used to generate the composite representationmay be recorded preoperatively or intra-operatively during a clinicalprocedure. In an alternative embodiment, a virtual visualization systemmay use standard representations (i.e., not patient specific) or hybridsof a standard representation and patient specific data. The compositerepresentation and any virtual images generated by the compositerepresentation may represent the static posture of a deformable anatomicregion during one or more phases of motion (e.g., during aninspiration/expiration cycle of a lung).

During a virtual navigation procedure, the sensor system 108 may be usedto compute an approximate location of the instrument with respect to thepatient anatomy. The location can be used to produce both macro-leveltracking images of the patient anatomy and virtual internal images ofthe patient anatomy. Various systems for using fiber optic sensors toregister and display a medical implement together with preoperativelyrecorded surgical images, such as those from a virtual visualizationsystem, are known. For example U.S. patent application Ser. No.13/107,562 (filed May 13, 2011) (disclosing “Medical System ProvidingDynamic Registration of a Model of an Anatomical Structure forImage-Guided Surgery”) which is incorporated by reference herein in itsentirety, discloses one such system.

The teleoperational medical system 100 may further include optionaloperation and support systems (not shown) such as illumination systems,steering control systems, irrigation systems, and/or suction systems. Inalternative embodiments, the teleoperational system may include morethan one teleoperational assembly and/or more than one operator inputsystem. The exact number of manipulator assemblies will depend on thesurgical procedure and the space constraints within the operating room,among other factors. The operator input systems may be collocated, orthey may be positioned in separate locations. Multiple operator inputsystems allow more than one operator to control one or more manipulatorassemblies in various combinations.

FIG. 2 illustrates a tracked instrument system 118 which includes themedical instrument system 104 and its interfacing systems. The medicalinstrument system 104 includes a flexible instrument 120 coupled by aninterface 122 to the teleoperational assembly 102 and the visualizationsystem 110. The instrument 120 has a flexible body 124, a tip 126 at itsdistal end 128, and the interface 122 at its proximal end 130. The body124 houses cables, linkages, or other steering controls (not shown) thatextend between the interface 122 and the tip 126 to controllably bend orturn the tip as shown for example by the dotted line versions of thebent tip 126, and in some embodiments control an optional end effector132. The flexible instrument may be steerable, including the steeringcontrols previously described, or may be non-steerable with nointegrated mechanism for operator control of the instrument bending. Theend effector may be a working distal part that is manipulable for amedical function, e.g., for effecting a predetermined treatment of atarget tissue. For instance, some end effectors have a single workingmember such as a scalpel, a blade, an endoscope, or an electrode. Otherend effectors such as shown in the embodiment of FIG. 2 , have a pair orplurality of working members such as forceps, graspers, scissors, orclip appliers, for example. Examples of electrically activated endeffectors include electrosurgical electrodes, transducers, sensors, andthe like. End effectors may also include conduits to convey fluids,gases or solids to perform, for example, suction, insufflation,irrigation, treatments requiring fluid delivery, accessory introduction,biopsy extraction and the like). In other embodiments, flexible body 124can define one or more lumens through which surgical instruments can bedeployed and used at a target surgical location. In various embodiments,the instrument 120 may include a flexible bronchial instrument, such asa bronchoscope or bronchial catheter for use in examination, diagnosis,biopsy, or treatment of a lung. The flexible body is also suited fornavigation and treatment of other tissues, via natural or surgicallycreated connected passageways, in any of a variety of anatomical systemsincluding the colon, the intestines, the kidneys, the brain, the heart,the circulatory system, or the like.

The instrument 120 can also include an image capture element 134 whichmay include a stereoscopic or monoscopic camera disposed at the distalend 128 for capturing images that are transmitted to and processed bythe visualization system 110 for display by the display system 111.Alternatively, the image capture element 134 may be a coherentfiber-optic bundle that couples to an imaging and processing system onthe proximal end of the instrument 120, such as a fiberscope. The imagecapture element 134 may be single or multi-spectral for capturing imagedata in the visible or infrared/ultraviolet spectrum.

In the pictured embodiment, the tracking system 135 includes anelectromagnetic (EM) sensor system 136 and a shape sensor system 138 fordetermining the position, orientation, speed, pose, and/or shape of thedistal end 128 and of one or more segments 137 along the instrument 120.Although only an exemplary set of segments 137 are depicted in FIG. 2 ,the entire length of the instrument 120, between the distal end 128 andthe proximal end 130 (and including the tip 126) may be effectivelydivided into segments. The tracking system 135 may be implemented ashardware, firmware, software or a combination thereof which interactwith or are otherwise executed by one or more computer processors, whichmay include the processors of a control system 116.

The EM sensor system 136 includes one or more conductive coils that maybe subjected to an externally generated electromagnetic field. Each coilof the EM sensor system 136 then produces an induced electrical signalhaving characteristics that depend on the position and orientation ofthe coil relative to the externally generated electromagnetic field. Inone embodiment, the EM sensor system may be configured and positioned tomeasure six degrees of freedom, e.g., three position coordinates X, Y, Zand three orientation angles indicating pitch, yaw, and roll of a basepoint. Further description of an EM sensor system is provided in U.S.Pat. No. 6,380,732, filed Aug. 11, 1999, disclosing “Six-Degree ofFreedom Tracking System Having a Passive Transponder on the Object BeingTracked,” which is incorporated by reference herein in its entirety.

The sensor system 138 includes an optical fiber 140 aligned with theflexible body 124 (e.g., provided within an interior channel (not shown)or mounted externally). The tracking system 135 is coupled to a proximalend of the optical fiber 140. In this embodiment, the fiber 140 has adiameter of approximately 200 tun. In other embodiments, the dimensionsmay be larger or smaller.

The optical fiber 140 forms a fiber optic bend sensor for determiningthe shape of the instrument 120. In one alternative, optical fibersincluding Fiber Bragg Gratings (FBGs) arc used to provide strainmeasurements in structures in one or more dimensions. Various systemsand methods for monitoring the shape and relative position of a opticalfiber in three dimensions are described in U.S. patent application Ser.No. 11/180,389, filed Jul, 13, 2005, disclosing “Fiber optic positionand shape sensing device and method relating thereto;” U.S. ProvisionalPat. App. No. 60/588,336, filed on Jul. 16, 2004, disclosing“Fiber-optic shape and relative position sensing;” and U.S. Pat. No.6,389,187, filed on Jun. 17, 1998, disclosing “Optical Fibre BendSensor,” which are incorporated by reference herein in their entireties.In other alternatives, sensors employing other strain sensing techniquessuch as Rayleigh scattering, Raman scattering, Brillouin scattering, andfluorescence scattering may be suitable.

In other alternative embodiments, the shape of the instrument 120 may bedetermined using other techniques. For example, if the history ofinstrument tip's pose is stored for an interval of time that is smallerthan the period for refreshing the navigation display or for alternatingmotion (e.g., inhalation and exhalation), the pose history can be usedto reconstruct the shape of the device over the interval of time. Asanother example, historical pose, position, or orientation data may bestored for a known point of an instrument along a cycle of alternatingmotion, such as breathing. This stored data may be used to develop shapeinformation about the instrument. Alternatively, a series of positionalsensors, such as EM sensors, positioned along the instrument can be usedfor shape sensing. Alternatively, a history of data from a positionalsensor, such as an EM sensor, on the instrument during a procedure maybe used to represent the shape of the instrument, particularly if ananatomical passageway is generally static. Alternatively, a wirelessdevice with position or orientation controlled by an external magneticfield may be used for shape sensing. The history of its position may beused to determine a shape for the navigated passageways.

In this embodiment, the optical fiber 140 may include multiple coreswithin a single cladding 146. Each core may be single-mode withsufficient distance and cladding separating the cores such that thelight in each core does not interact significantly with the lightcarried in other cores. In other embodiments, the number of cores mayvary or each core may be contained in a separate optical fiber.

In some embodiments, an array of FBG's is provided within each core.Each H3G comprises a series of modulations of the core's refractiveindex so as to generate a spatial periodicity in the refraction index.The spacing may be chosen so that the partial reflections from eachindex change add coherently for a narrow band of wavelengths, andtherefore reflect only this narrow band of wavelengths while passingthrough a much broader band. During fabrication of the FBG's, themodulations arc spaced by a known distance, thereby causing reflectionof a known band of wavelengths. However, when a strain is induced on thefiber core, the spacing of the modulations will change, depending on theamount of strain in the core. Alternatively, backscatter or otheroptical phenomena that vary with bending of the optical fiber can beused to determine strain within each core.

Thus, to measure strain, light is sent down the fiber, andcharacteristics of the returning light are measured. For example, FBG'sproduce a reflected wavelength that is a function of the strain on thefiber and its temperature. This FBG technology is commercially availablefrom a variety of sources, such as Smart Fibres Ltd. of Bracknell,England. Use of FBG technology in position sensors for teleoperationalsurgery is described in U.S. Pat. No. 7,930,065, filed Jul. 20, 2006,disclosing “Teleoperational Surgery System Including Position SensorsUsing Fiber Bragg Gratings,” which is incorporated by reference hereinin its entirety.

When applied to a multicore fiber, bending of the optical fiber inducesstrain on the cores that can be measured by monitoring the wavelengthshifts in each core. By having two or more cores disposed off-axis inthe fiber, bending of the fiber induces different strains on each of thecores. These strains are a function of the local degree of bending ofthe fiber. For example, regions of the cores containing FBG's, iflocated at points where the fiber is bent, can thereby be used todetermine the amount of bending at those points. These data, combinedwith the known spacings of the FBG regions, can be used to reconstructthe shape of the fiber. Such a system has been described by LunaInnovations, Inc. of Blacksburg, Va.

As described, the optical fiber 140 is used to monitor the shape of atleast a portion of the instrument 120. More specifically, light passingthrough the optical fiber 140 is processed by the tracking system 135for detecting the shape of the surgical instrument 120 and for utilizingthat information to assist in surgical procedures. The tracking system135 may include a detection system for generating and detecting thelight used for determining the shape of the instrument 120. Thisinformation, in turn, in can be used to determine other relatedvariables, such as velocity and acceleration of the parts of a surgicalinstrument. By obtaining accurate measurements of one or more of thesevariables in real time, the controller can improve the accuracy of theteleoperational surgical system and compensate for errors introduced indriving the component parts. The sensing may be limited only to thedegrees of freedom that are actuated by the teleoperational system, ormay be applied to both passive (e.g., unactuated bending of the rigidmembers between joints) and active (e.g., actuated movement of theinstrument) degrees of freedom.

The information from the tracking system 135 may be sent to thenavigation system 142 where it is combined with information from thevisualization system 110 and/or the preoperatively taken images toprovide the surgeon or other operator with real-time positioninformation on the display system 111 for use in the control of theinstrument 120. The control system 116 may utilize the positioninformation as feedback for positioning the instrument 120. Varioussystems for using fiber optic sensors to register and display a surgicalinstrument with surgical images are provided in U.S. patent applicationSer. No. 13/107,562, filed May 13, 2011, disclosing, “Medical SystemProviding Dynamic Registration of a Model of an Anatomical Structure forImage-Guided Surgery,” which is incorporated by reference herein in itsentirety.

In the embodiment of FIG. 2 , the instrument 104 is teleoperated withinthe teleoperational surgical system 100. In an alternative embodiment,the teleoperational assembly 102 may be replaced by direct operatorcontrol. In the direct operation alternative, various handles andoperator interfaces may be included for hand-held operation of theinstrument.

FIG. 3A depicts a composite image 150 including an image 151 of a humanlung 152, from a viewpoint external to the lung, registered with aninstrument image 154 of a flexible instrument, such as the flexibleinstrument 120. The image 151 of the lung 152 may be generated frompreoperatively recorded images or may be generated concurrently duringthe surgical procedure. The composite image 150 may be displayed viadisplay system 111. As the instrument 120 is advanced through bronchialpassageways 156 of the lung 152, information from the tracking system135 and/or the visualization system 110 is used to register theinstrument image 154 with the lung image 151. The image 151 of the lung152 may change, for example, to depict the lung in a state ofinspiration or expiration. The instrument image 154 may change to depictthe advancement or withdrawal of the instrument 120 through thebronchial passageways 156. Occasionally, the composite image 150 mayerroneously render the instrument image 154 such that a portion of theinstrument image 154′ is outside of a bronchial passageway (as shown inFIG. 3A) or is positioned within an incorrect bronchial passageway (asshown in FIG. 3B). Systems and methods arc described below forcorrecting the instrument image such that the instrument is locatedwithin the correct bronchial passageway.

FIG. 3C is an internal image 160 of the human lung 152 depicting aregion of the lung from the viewpoint of the instrument 120. The image160 may be a concurrent image taken during the surgical procedure by theinstrument 120 while located in the depicted portion of the lung 152.More specifically, the image may be captured by the visualization system110. Alternatively, the image 160 may be a preoperatively recorded imageselected based upon the location of the tip of the instrument 120 asdetermined by the tracking system 135.

Returning to FIG. 2 , the tracking system 135, which includes the EMsensor system 136 and the shape sensor system 138, may calculate aposition for the instrument tip 126 or one or more segments 137 of theinstrument 120 indicating that the tip is outside the actual anatomicalpassageway containing the relevant instrument parts (e.g., as beingpositioned outside the bronchial tree entirely or within the wrongpassageway). This likely indicates a slight measurement error (assumingthe wall of the anatomical passageway has not been breached). Such anerror may result from the dynamic nature of certain anatomic structuressuch as the lungs or the heart. For example, inhalation and exhalationchanges the position and size of the bronchial passageways of the lung.Alternatively, the error may result from patient motion or from tissuedeformation caused by the presence of the surgical instrument within theanatomic passageways. In some instances, the error may result fromtissue deformation caused by fluid accumulation or tumor growth (e.g.,progressive tumor growth since the pre-operative imaging was obtained).

To correct the position of the instrument and accurately locate one ormore points of the instrument within the correct passageway when theimage of the instrument and the image of the patient anatomy areco-registered and displayed, selected points of the instrument may besnapped or graphically registered to a location on the wall of theanatomical passageway or to the lumen of the anatomical passageway. Aswill be described in detail below, a variety of compensation methods maybe used to correct the model of the patient anatomy to adjust forinternal and external deformation forces, patient movement, or otherchanges in the patient anatomy subsequent to the recording of thepreoperative image. Internal deformation forces on tissues of theanatomy may result, for example, from movement between breathing statesof inspiration and expiration, cardiac movement, tumor growth, fluidaccumulation, and coughing. External deformation threes on tissues ofthe anatomy may result, for example, from instrument insertion andmanipulation. A corrected model of the patient anatomy allows for thegeneration of a more accurate composite image of the instrument relativeto the anatomical passageways.

A corrected model may be particularly useful in the context of virtualnavigation inside the anatomical passageways of a patient. Virtualnavigation is based upon reference to a pre-operatively acquired datasetassociated with the three dimensional anatomical structure of thepassageways. For example, the dataset may be acquired by a preoperativeCT scan. Software is used to convert the CT images into a threedimensional model describing the various locations and shapes of thepassageways and their connectivity. During the virtual navigationprocedure, the sensor system, particularly the EM sensor system, may beused to compute an approximate location of the instrument with respectto the patient anatomy. Typically an assumption is made that all partsof the patient anatomy are fixed with respect to each other. Under thisassumption, a virtual view from the location of the tip of theinstrument inside the patient anatomy, as shown in FIG. 3C, can becomputed from the preoperative CT scan dataset.

As previously described, the assumption that the patient anatomy remainsfixed is generally invalid due to various deformation forces (e.g., bothinternal and external) applied to the patient anatomy. To compensate forthe motion or other errors introduced by the sensor system, a virtualview may be generated from the closest point to the sensed location ofthe tip of the instrument, inside the passageway, rather than from thecomputed position of the tip of the instrument. The process of adjustingthe location of the sensed location of the instrument to an adjustedlocation within or on the wall of a passageway is known as snapping.Snapping may work particularly well when passageways are well separatedand not densely packed. When deformation forces are large and thepassageways are dense (and especially when the passageways have similarcurvatures), snapping alone may result in an incorrect choice for theparticular airway passage in which the instrument is located.

FIG. 4A is an illustration of an undeformed model 200 of a bronchialpassageway 202 of a lung. A catheter 204 extends into the bronchialpassageway 202. The catheter 204 may be substantially similar to theinstrument 120. The model 2(X) may be displayed on a display system ormay reside in an undisplayed form in a computer memory. As shown, thecatheter 204 exerts deformation forces F on the bronchial passageway202.

FIG. 4B illustrates the effect of the deformation forces F on the model200. The forces F of the catheter 204 move the bronchial passage 202from an initial position illustrated by the model 200 to a new positionillustrated by a deformed model 206. Thus, the deformed model 206illustrates the effect of the forces of the catheter 164 on location andorientation of the bronchial passageway 202.

FIG. 5A illustrates a composite image of a sensed model 250 of acatheter 252 superimposed on an undeformed model 254 of a branchedstructure of bronchial passageways 256 of a lung. The bronchialpassageways 256 of the lung include the distinct passageways 257, 258,259, and 260. Although the catheter 252 is known to extend into one ofthe bronchial passageways 257, 258, 259, and 260 (for example through aninternal image obtained by the catheter 252 as in FIG. 3C), the sensedmodel 250 depicts the catheter 252 as positioned outside the passageways257, 258, 259, and 260. The passageways 257, 258, 259, and 260 may beseparated from each other by only a few millimeters, and the passageways257, 258, 259, and 260 may have very similar curvatures, as shown inFIG. 5A. Merely matching the sensed shape of the catheter 252 to themodeled shape of one of the passageways 257, 258, 259, and 260A may notbe sufficient to determine which passageway has actually received thecatheter 252. To more accurately determine which passageway has receivedthe catheter 252, a deformed passageway model may be used that takesinto consideration the forces exerted on the passageway by varyingregions of stiffness of the catheter. This deformed passageway modelwill depict the effect of the varying forces of the catheter 252 on thelocation and orientation of the bronchial passageways 256, and therebycorrectly depict the position of the catheter 252 relative to thepassageways 256.

As will be described in detail below, a process for registering thecatheter 252 to an anatomic passageway takes into consideration thedeformation forces exerted on the passageway tissue by the activeportions of the catheter (i.e., portions under direct operator control)as well as the variably passive portions of the catheter. The processfor registration also takes into consideration the anatomicalrestorative forces associated with the flexibility/rigidity of thepassageway tissue and the surrounding tissue. One or more candidateanatomical passageways may be modeled to match the sensed catheter shapeand/or tip position. The candidate anatomical passageway(s) may then beevaluated in view of the deformation and restorative forces to determinethe most likely matching passageway for registration. This evaluationprocess includes eliminating any candidate passageways that would not beable to achieve the deformation, as modeled, from the combination ofdeformation and restorative forces. Alternatively, the candidateanatomical passageways may be modeled to reflect the deformation andrestorative forces, and the deformed models may be compared to thecatheter shape to identify a matched passageway for registration.Optionally, the model of the matched passageway, as altered by thecombination of the deformation and restorative forces, is displayed.Optionally, an image of the catheter may be illustrated in registrationwith the deformed matching passageway.

Generally, the catheter 252 does not include uniform physical propertiesalong its inserted length. For example, in the pictured embodiment, thecatheter 252 includes 3 distinct portions having variable physicalproperties. In particular, the catheter 252 includes a first section 262extending from A to B, a second section 264 extending from B to C, and athird section 266 extending from C to D. The catheter 252 has variablestiffness along its length. For example, the first section 262 has agreater stiffness than the second section 264, and the third section 266comprises an active section having actively controllable stiffness(e.g., upon active steering of the third section 266). In someembodiments, the catheter 252 has a gradually tapered stiffnessextending along its length from A to D. Assuming the third section 266to be the active section, as in the pictured embodiment, the firstsection 262 may have the tendency to lay on the supporting structure ofthe passageway 256 while remaining as straight as possible, the secondsection 264 may have a tendency to comply with the pose of the firstsection 262 and the surrounding tissue, and the third section 266 maycause local deformation only (e.g., along the length of the thirdsection 266 only). In other words, the deformation forces exerted by thethree separate sections 262, 264, and 266 are not necessarily the sameor equal. The force exerted on the bronchial passageways 256 by thethird section 266 may be significantly stronger than the forces exertedon the bronchial passageways 256 by the sections 262 and 264. At thesame time, the force exerted by the third section 266 may be constrainedto a smaller area than the forces exerted on the bronchial passageways256 by the sections 262 and 264. To evaluate candidate matchingpassageways and to construct an accurate deformed model of the bronchialpassageways 256 showing the correct positioning of the catheter 252within the correct passageway, the effect of the individual sections ofthe catheter 252 on the separate portions of the bronchial passagewaysmust be determined.

FIGS. 5 .B-5D illustrate composite images of the sensed model 250 of thecatheter 252 (e.g., based on shape sensor data or cumulative positionsensor data) positioned within different passageways of deformedcandidate models of the bronchial passageways 256. FIG. 5B illustrates acomposite image showing the catheter 252 positioned within thepassageway 257′ of a deformed model 270 of the bronchial passageways256. The deformed model 270 assumes that the forces exerted by theseparate sections of the catheter 252 caused the passageways 256 toshift downwards such that a lower border 272 of the passageway 257 isshifted downwards by a distance D1 (to appear as the passageway 257′with the lower border 272′ in the deformed model 270). If, however, theforces exerted by the separate sections of the catheter 252 and theknown tissue restorative forces would not have caused the passageways256 to shift downwards by a distance D1, the deformed model 270 may beeliminated as a candidate. In other words, in view of the physicalproperties of the catheter 252 (and in particular the second section 264and/or the third section 266) and the physical properties of thesurrounding tissue, if the catheter 252 would not apply sufficient forceto displace the bronchial passageway 257 by the distance D1, the controlsystem 116 (or another part of the system 100) can determine that thehypothetical deformed model 270 is an unlikely candidate forregistration with the known shape of the catheter.

FIG. 5C illustrates a composite image showing the catheter 252positioned within the passageway 258′ of a deformed model 280 of thebronchial passageways 256. The deformed model 280 assumes that theforces exerted by the separate sections of the catheter 252 caused thepassageways 256 to shill downwards such that a lower border 282 of thepassageway 258 is shifted downwards by a distance D2 (to appear as thepassageway 258′ with the lower border 282′ in the deformed model 280).If, however, the forces exerted by the separate sections of the catheter252 and the known tissue restorative forces would not have caused thepassageways 256 to shift downwards by a distance D2, the deformed model280 may be eliminated as a candidate. In other words, in view of thephysical properties of the catheter 252 (and in particular the secondsection 264 and/or the third section 266) and the physical properties ofthe surrounding tissue, if the catheter 252 would not apply sufficientforce to displace the bronchial passageway 257 by the distance D2, thecontrol system 116 (or another part of the system 100) can determinethat the hypothetical deformed model 280 is an unlikely candidate forregistration with the known shape of the catheter.

FIG. 5D illustrates a composite image showing the catheter 252positioned within the passageway 259′ of a deformed model 290 of thebronchial passageways 256. The deformed model 290, assumes that theforces exerted by the separate sections of the catheter 252 caused thepassageways 259, 260 to shift upwards such that an upper border 292 ofthe passageway 259 is shifted upwards by a distance D3 (to appear as thepassageway 259′ with the upper border 292′ in the deformed model 290).If, however, the forces exerted by the separate sections of the catheter252 and the known tissue restorative forces would not have caused thepassageway 259 to shift upward by a distance D3, the deformed model 290may be eliminated as a candidate. In other words, in view of thephysical properties of the catheter 252 (and in particular the secondsection 264 and/or the third section 266) and the physical properties ofthe surrounding tissue, if the catheter 252 would not apply sufficientforce to displace the bronchial passageway 259 upward by the distanceD3, the control system 116 (or another part of the system 100) candetermine that the hypothetical deformed model 290 is an unlikelycandidate for registration with the known shape of the catheter.

After the unlikely candidate models are eliminated, the remainingcandidate models may be evaluated to determine which one is the mostlikely model to match the sensed catheter, based upon the deformationand restorative forces. Although FIGS. 5B-5D illustrate the candidatematching process, it is understood that the process may be conducted bythe control system 116 without display of each of the candidatedeformation models. When a deformed model is finally selected forregistration with the catheter, the deformed model and registeredcatheter may be displayed. In various alternative embodiments, thecandidate matching process may be illustrated for display.

FIG. 6 illustrates a diagrammatic representation of a sensed shape ormodel 293 of the catheter 252 shown in FIGS. 5A-5D in comparison withtwo different candidate models 294 and 295 representing two possiblepassageways that may contain the catheter 252. As shown, merelycomparing the sensed shape of the catheter 252 may not be adequate todetermine the correct passageway because the two models 294 and 295 havesubstantially similar curvatures. In particular, the curvature of thesensed shape 293 of the catheter 252 extending between points 296 and297 is substantially similar to the curvature of the model 294 extendingbetween points 296 and 298, as well as the curvature of the model 295extending between points 296 and 299. Thus, to determine the correctposition of the catheter 252, the control system 116 can compare (e.g.,using one or more algorithms of the control system 116) the sensed shape293 of the catheter 252 with the candidate models 294 and 295, and alsofactor in the physical characteristics of the different sections of thecatheter 252 (e.g., the sections 262, 264. and 266) and the physicalcharacteristics of the tissue surrounding the different sections of thecatheter 252. The physical characteristics of the catheter 252associated with deformation forces include, by way of non-limitingexample, the degree of stiffness of the different sections, the forceapplied by each section (and in particular the active section at the tipor section 266), the direction of the forces applied by each section,and the material composition and weight of the catheter sections. Thephysical characteristics of the tissue associated with restorationforces include, by way of non-limiting example, the tissue type, thecomposition of the tissue (e.g., including the fluid or air content ofthe tissue), the compliance of the tissue, and the adjacent supportingtissue (e.g., bone, muscle, and/or vessels).

By employing such a comprehensive comparison, the control system 116 candetermine the correct passageway in which the catheter 252 (e.g., thedistal section 266 of the catheter 252) resides. For example, if, basedupon the known deformation and restorative forces, the catheter 252would not be able to shift the point 299 of the model 295 downward by adistance D4 to the point 297 or that the catheter 252 would not be ableto shift the model 295 laterally by a distance D5 to the point 297, thenthe control system 116 may conclude that the candidate model 295 is anunlikely passageway to contain the catheter (as well as any otherpassageways located above or more lateral to the passageway representedby the model 295). Similarly, if the catheter 252 would not be able toshift the point 298 of the model 294 downward by a distance D6 to thepoint 297 or that the catheter 252 would not be able to shift the model295 laterally by a distance D7 to the point 297, then the control system116 may conclude that the model 294 is an unlikely passageway to containthe catheter (as well as any other passageways located above or morelateral to the passageway represented by the model 294). Aftereliminating candidate models, a remaining candidate model that conformsto the known deformation and restoration forces may be selected forregistration with an image of the catheter. Because the sensed model 293of the catheter 252 may be inaccurate to a certain degree, the controlsystem 116 may evaluate these sensor values within a pre-defined rangeof possibilities to account for such inaccuracies.

FIG. 7A illustrates a composite image of an undeformed model 300 of abranched structure of bronchial passageways and a catheter 302 extendingthrough the passageways. With the undeformed model 300, the catheter 302is shown extending outside of the passageways and a tip 304 of thecatheter 302 appears, in the composite image, to be located within thebronchial passageway L5. The catheter 302 may be the same as thecatheter 252 shown in FIGS. 5A-5D. For example, in the picturedembodiment, the catheter 302 includes a first section 305 extendingbetween the points A and B, a second section 306 extending between thepoints B and C, and a third section 307 extending between the points Cand D.

As shown in FIG. 7A, the bronchial passageways are modeled as a set ofrigid links L1-L10 connected by joints that can rotate arounddeformation points such as connection points C in both pitch and yawdirections. In some embodiments, other degrees of freedom such asstretch and roll may be accommodated. In other embodiments, joints orother types of deformation points may be located along the lengths ofthe links L1-L10) to allow multiple locations for bending along thelength of the links. In some embodiments, the deformation points may bedistributed throughout the modeled links L1-L10, without regard to thelocation of connection points. The tissue types and the tissuecomposition of the links and their immediately surrounding tissue canvary significantly. Thus, the individual links may have differentphysical properties that respond differently to the forces applied bythe catheter 302. The curvature of the links L3-L4 and the curvature ofthe links L8-L9 arc substantially similar. From the undeformed model300, it is unclear whether the sections 306 and 307 of the catheter 302extend into the links L3-L4 or the links L8-L9. In order to determinewhether the sections 306 and 307 of the catheter 302 extend into thelinks L3-L4 or the links L8-L9, the control system 116 can use theprocess discussed above in relation to FIGS. 5A-6 to evaluate thepossible deformation models in light of the physical properties of thedifferent catheter sections in addition to the physical properties ofthe tissue with which the catheter interacts.

To deform the model 300 and correct for the forces applied by thecatheter, the shape of the whole catheter in the bronchial passagewaysis determined. A shape sensor system, such as the shape sensor system138 may be used to determine the shape of the entire catheter, not justthe tip. The shape of the catheter is used to determine a set ofdeformation variables (e.g., pitch and yaw angles) at each connectionpoint C in the undeformed model. The methods used to convert the modeledpitch and yaw angles of the connection points to the poses of the linksin the bronchial structure are standard kinematic methods found, forexample in Siciliano et al., Springer Handbook of Teleoperationals(Springer, 2008). The amount of deformation of the bronchial passagewaysis determined by virtual forces caused by points of the catheterdepicted outside the passageways in the undeformed model as well as theopposing forces caused by the anatomy itself (e.g., the links of thebronchial passageways and their surrounding tissue). The catheter forcesact differently along the length of the catheter (e.g., different forcesare exerted by the section s 305, 306, and 307) and effectively act toshift the passageway in the direction of the catheter. The amount ofdeformation is also determined by the opposing or counteractingrestorative forces, such as the rigidity of the bronchial passageway andthe surrounding connective tissue, which bias the passageway toward theundeformed shape and configuration.

FIGS. 7B-7D depict the undeformed model 300 of the branched bronchialpassageways and also show a candidate deformation model 310 of thebranched bronchial passageways as the catheter 302 is advanced throughthe passageways. In FIG. 7B, the catheter 302 is advanced through linkL1 and into link L2. The deformation model 310 is shifted slightly fromthe undeformed model 300 to adjust for the direct forces of the section307 of the catheter 302 on the links L1, L2 and indirect forces on theconnected distal links L3-L10, as well to adjust for the opposing forcesof the tissue at links L1 and L2 (and, to a lesser extent, the moredistal links). The tip 304 is depicted as within the link L2 in both theundeformed model 300 and the deformed model 310, but the exact locationof the tip 304 within the link L2 is different between the two models.

In FIG. 7C, the catheter 302 is advanced through link L2 and into linkL3. The candidate deformation model 310 is further shifted from theundeformed model 300 to adjust for the direct forces of the section 306of the catheter 302 on the links L1, L2, L3, the direct forces of thesection 307 of the catheter 302 on the link L3, and indirect forces onthe connected distal links L4-L10, as well to adjust for the opposingforces of the tissue at links L1-L3 (and, to a lesser extent, the moredistal links). The tip 304 is depicted as outside the link 13 in theundeformed model 3(X) but within the link L3 in the deformed model 310.Thus, the deformed model 310 more accurately reflects the reality thatthe tip 304 is inside the bronchial passageway.

If, however, the link L3 would not be displaced to where the tip 304 issensed (within a range of sensor inaccuracy), then the control system116 may conclude that the candidate deformed model 310 may beeliminated. For example, based on the physical properties of thesections 306 and 307 of the catheter 302, the forces exerted by thosesections, and the tissue characteristics and opposing forces of thelinks L6-L8, the possibility of the tip 304 of the catheter 302extending into the links L6 and L8 may be evaluated instead. Inperforming such an evaluation, the control system 116 may constructanother candidate deformation model for the catheter 302 extending intothe links L6 and L8. In some embodiments, the control system 116constructs both possible models (e.g., one showing the catheter 302lying within links L3 and L4 and another showing the catheter 302 lyingwithin the links L6 and L8) immediately after the recognizing that bothmodels are candidates based upon the shape of the catheter 302. In otherembodiments, the control system 116 initially constructs the deformedmodel that appears most likely (e.g., based on curvature comparisonsand/or lines of best-fit between the catheter and the passageways),evaluates this model for accuracy, and only constructs the second modelif this model is determined to have a certain degree of error oruncertainty below a predefined threshold (e.g., less than a 5% chance oferror).

In FIG. 7D, the catheter 302 is advanced into link L4. The deformationmodel 310 is further shifted from the undeformed model 3(X) to adjustfor the direct forces of the catheter 302 on the links L1, L2, L3, L4and indirect forces on the connected distal links L5-L10. The tip 304 islocated in link L5 in the undeformed model 300 and inside the link L4 inthe deformed model 310. The control system 116 can confirm that thedeformed model 310 is an accurate depiction of the positioning of thecatheter 302. Based on the forces applied by each individual section ofthe catheter 302 on the different links of the bronchial passageways,and the opposing forces applied by each link of the bronchialpassageways, the degree of accuracy of the deformed model may bedetermined. If the deformed model is determined to be accurate and acalculated chance of inaccuracy is below a predefined threshold oferror, the control system 116 can conclude that the deformed model 310more accurately reflects the actual location of the tip 304. The finaldeformed model 310 may be displayed as depicted in FIG. 8 . It isunderstood that the candidate model evaluation process may be performedby the control system 116 without display until the matched model isdetermined.

One embodiment of a method for correcting an undeformed model into amore accurate deformed model is provided in the flowchart 400 of FIG. 9. The model may be generated from stored data describing the patientanatomy, for example, from preoperative images, from two or threedimensional datasets describing the patient anatomy, from a standard(i.e., not patient specific model), or from hybrids of any of the above.Initially the deformed model is equal to the static, undeformed modelwith all deformation variables (e.g., pitch/yaw angles, stretch) set atzero. At step 402, the current shape of a catheter of a surgicalinstrument or another type of sensed device is acquired using one ormore sensors of the sensor system. At step 404, a sampling of pointsalong the shaped catheter relative to a sampling of points along thestatic model is matched for a best-fit. In greater detail, this best-fitmatching technique may, in one example embodiment, include a) finding Nlinks L that are closest to the tip of the catheter; b) for each of theN links L, computing the unique subset of links that form a path fromthat link to the base of the bronchial structure; c) for each of the Nsubsets, computing the closest points on the bronchial passageway toeach of the sampled points along the catheter shape; d) computing adistance score for each of the N options as a function of the distancesof the sampled points to the corresponding closest points in thebronchial passageway; and choosing the best option from the N optionsbased on the lowest combined distance score. Thus, the most likelycandidate passageway (or combination of anatomical links) is chosenbased on a comparison of the curvature of the catheter and the curvatureof the different passageways. Optionally, a plurality of candidatemodels may be determined.

At step 406, a set of deformation forces for the selected best fit(e.g., the selected passageway) model(s) are calculated. For each of thesampled points along the catheter shape, a force is defined as thevector between the position of the point and the position of the closestpoint on the bronchial passageway. As mentioned above, the cathetergenerally will include distinct sections having different physicalproperties, and the algorithm individually evaluates the forces appliedby each distinct section based on its particular physical propertiesrelative to its location within the patient anatomy (e.g., in realtime). For example, the forces applied by a distal active section (e.g.,the section 307 in FIG. 7C) may be greater and more deformative on thesurrounding tissues than an inactive or more flexible section of thecatheter (e.g., the section 306 in FIG. 7C). Thus, device-awarecalculations are performed to accurately evaluate the deformation forcesapplied by different parts of the catheter.

At step 408, a set of restorative forces are computed that correspond tothe forces that bias the bronchial passageway toward its original shape.These restorative forces are defined between sampled point on thedeformed bronchial passageway links and corresponding points on theundeformed links. These restorative forces may depend upon a variety offactors, including by way of non-limiting example, the type of tissue,the fluid/air content of the tissue, the neighboring or supportingtissue, the mass of the tissue, the elasticity of the tissue, and thedensity of the tissue. Optionally, a subset of the restorative forcesknown as maintenance forces may be computed that correspond to theforces that bias the local shape to the undeformed local shape. Thesemaintenance forces can be defined, for example, as joint torques actingbetween each pair of connected links with torque equal and opposite tothe deformation angle at that connection. The total set of forces actingon the branched structure of bronchial passageways is computed as aweighted sum of the sets of deforming and restorative forces.

Different weight or weighting factors may be determined by a user orpreset by the control system 116 in a way that results in the mostrealistic behavior, for example based on the known physical propertiesof the distinct sections of the catheter and the measured mechanicalproperties of the anatomy. The tissue's elastostatic properties can alsobe pre-estimated in multi- dimensional diagnostic images (e.g., 3D or 4DCT) possibly also involving real patient image databases for intensityto elasticity calibration. Since the CT imaging modality capturesrelative tissue density, a mapping between the image intensities and atissue site's elastic properties can be pre-computed. During run time,tissue deformation under forces of known magnitude and direction foreach distinct section of the catheter can be computed to reflect morephysically accurate tissue behavior.

At step 410, one or more candidate deformation models of the bronchialpassageways are created by adjusting each joint between the bronchialpassageway links by an amount proportional to the total applied forceand/or torque at the joint. Thus, the deformation model is created basedupon both deformation forces associated with the physical properties ofthe distinct catheter sections and the restorative forces associatedwith the physical characteristics of the tissue affected by thesecatheter sections to refine the initial best-fit passagewayselection(s). The candidate deformation models are compared to thecurrent shape of the instrument body to determine which passageway bestmatches with the current shape of the instrument body. Alternatively,the candidate passageways are modeled to match the catheter shape and anevaluation is performed to determine which of the deformed modelsmatching the catheter shape are most likely based upon the knowndeforming and restorative forces.

At step 412, a composite image of the deformed model and the catheter isgenerated to accurately depict the location of the catheter within thebranched structure of the bronchial passageways. Additionally, with atip of the catheter registered to the correct link in the deformedmodel, a correct virtual navigation view can be retrieved from thepreoperatively stored images. The process may be repeated from step 402until the catheter reaches its desired target within the bronchialstructure.

In another embodiment, the non-static nature of the branched bronchialstructure can be shown in an inspiration model of the bronchialstructure that corresponds to an inhalation state of the bronchialstructure and an expiration model that corresponds to an exhalationstate of the bronchial structure. With reference to FIGS. 9 and 10 , thepose, position, or orientation of a catheter already adjusted based upona sensor system can be further refined or warped to the shape of abronchial passageway as the passageway adjusts between a state ofinspiration and expiration and vice versa. FIG. 9 depicts an inspirationmodel 450 and an expiration model 452. A catheter 454 is initiallylocated based upon the sensor system but is adjusted to remain withinthe image of the bronchial passageways as they move between states ofinspiration and expiration.

FIG. 10 is a flow chart 460 describing a method for adjusting thecatheter based on states of inspiration and expiration. At step 462, thecurrent shape of the catheter is acquired. At step 464, the cathetershape is matched to the shape of the bronchial passageway in a state ofinspiration. At step 466, the catheter shape is matched to the shape ofthe bronchial passageway in a state of expiration. In a composite imageof the bronchial passageways and the catheter, the image of the catheteris maintained within the bronchial passageways for each state ofexpiration and inspiration. The process may be repeated until thecatheter reaches its desired target within the bronchial structure. Inalternative embodiments, the process may be used to correct images forany instrument located within any moving anatomical structure, and inparticular an anatomic structure, such as the heart or lungs, thatrepeatedly alternates between known states. For example, the movinganatomical structure may be a heart alternating between diastole andsystole phases of the cardiac cycle.

In alternative embodiments, the instruments or the anatomic passagewaysmay be modeled using three dimensional shape histograms to performsimilarity searching. For example, a metric may be developed to measuresimilarity between two three-dimensional curves. With reference to FIG.11 , a curve 500 corresponds to a measured device path through areference anatomy and a curve 502 corresponds to an actual anatomicpassageway. For each point P_(i) on these curves, the followingparameters may be defined: a) local gradient, b) local curvature, c)radial angular displacement with respect to an endpoint connector, d)radial distance with respect to an endpoint connector, e) radialangular/displacements with respect to a third reference axis 504.

As shown in FIG. 12 , a histogram 505 may be created from each of theabove parameters. The histograms are matched directly to create a metricfor each curve similarity.

In another embodiment, an arbitrarily shaped point set is snapped toanother point set using a metric to measure the quality of the match.With reference to FIG. 13 , two point sets P_(T) and P_(S) are provided.Starting with P_(S), a similarity measure |S_(S→T)| is computed. Thesimilarity measure is a one dimensional histogram indexed on distancefrom a reference point. The value of a tangent vector at the referencepoint is stored with respect to a reference vector. An optimizer is usedto optimize the degrees of freedom on a transform to maximize thesimilarity measure at an instant between P_(S) and P_(T).

Alternative systems and methods for registering an image of at least aportion of a flexible instrument to an image of an anatomical system aredisclosed in U.S. patent application Ser. No. 13/893,040, entitled“Systems and Methods for Registration of a Medical Device Using aReduced Search Space” and in U.S. patent application Ser. No.13/892,924, entitled “Systems and Methods for Registration of a MedicalDevice Using Rapid Pose Search,” both of which are incorporated byreference herein in their entirety. Aspects of theses incorporated byreference systems and methods may be used together with the abovedisclosed systems and methods to provide alternative methods ofaccurately registering an image of at least a portion of a flexibleinstrument to an image of an anatomical system.

Although the registration systems and methods have been described hereinwith respect to teleoperated or hand operated surgical systems, theseregistration systems and methods will find application in a variety ofmedical and non-medical instruments in which accurate instrument imageregistration is otherwise too time consuming or computer processingintensive.

Although the systems and methods of this disclosure have beenillustrated for use in the connected bronchial passageways of the lung,they are also suited for navigation and treatment of other tissues, vianatural or surgically created connected passageways, in any of a varietyof anatomical systems including the colon, the intestines, the kidneys,the brain, the heart, the circulatory system, or the like. The methodsand embodiments of this disclosure are also suitable for non-surgicalapplications.

One or more elements in embodiments of the invention may be implementedin software to execute on a processor of a computer system such ascontrol system 116. When implemented in software, the elements of theembodiments of the invention are essentially the code segments toperform the necessary tasks. The program or code segments can be storedin a processor readable storage medium or device that may have beendownloaded by way of a computer data signal embodied in a carrier waveover a transmission medium or a communication link. The processorreadable storage device may include any medium that can storeinformation including an optical medium, semiconductor medium, andmagnetic medium. Processor readable storage device examples include anelectronic circuit; a semiconductor device, a semiconductor memorydevice, a read only memory (ROM), a flash memory, an erasableprogrammable read only memory (EPROM); a floppy diskette, a CD-ROM, anoptical disk, a hard disk, or other storage device, The code segmentsmay be downloaded via computer networks such as the Internet, Intranet,etc.

Note that the processes and displays presented may not inherently berelated to any particular computer or other apparatus. Variousgeneral-purpose systems may be used with programs in accordance with theteachings herein, or it may prove convenient to construct a morespecialized apparatus to perform the operations described. The requiredstructure for a variety of these systems will appear as elements in theclaims. In addition, the embodiments of the invention are not describedwith reference to any particular programming language. It will heappreciated that a variety of programming languages may be used toimplement the teachings of the invention as described herein.

While certain exemplary embodiments of the invention have been describedand shown in the accompanying drawings, it is to be understood that suchembodiments are merely illustrative of and not restrictive on the broadinvention, and that the embodiments of the invention not be limited tothe specific constructions and arrangements shown and described, sincevarious other modifications may occur to those ordinarily skilled in theart.

1-23. (canceled)
 24. A medical system comprising: a flexible device configured to be positioned at least partially within an anatomical passageway of a plurality of anatomical passageways within a patient anatomy; and a memory device including computer executable instructions, the computer executable instructions for performing operations comprising: determining a deformation force exerted by a section of the flexible device; and registering a model of a candidate anatomical passageway of the plurality of anatomical passageways to a model of the flexible device based on: a shape of the flexible device; and the deformation force exerted by the section of the flexible device.
 25. The medical system of claim 24, wherein the computer executable instructions further comprise instructions for identifying the candidate anatomical passageway in a model of the patient anatomy.
 26. The medical system of claim 25, wherein the model of the patient anatomy comprises an undeformed model of the patient anatomy including a model of the plurality of anatomical passageways.
 27. The medical system of claim 25, wherein the instructions for identifying the candidate anatomical passageway comprise instructions for matching a plurality of points of the model of the candidate anatomical passageway to a plurality of points of the model of the flexible device, the model of the flexible device based on the shape of the flexible device.
 28. The medical system of claim 24, wherein the instructions for identifying the candidate anatomical passageway comprise instructions for comparing the shape of the flexible device to a shape of a model of each anatomical passageway of the plurality of anatomical passageways.
 29. The medical system of claim 28, wherein the instructions for identifying the candidate anatomical passageway comprise instructions for eliminating at least one other anatomical passageway of the plurality of anatomical passageways from consideration based on the comparison of the shape of the flexible device to the shape of the model of each anatomical passageway of the plurality of anatomical passageways.
 30. The medical system of claim 24, wherein the section of the flexible device comprises a distal section of the flexible device.
 31. The medical system of claim 24, wherein the section of the flexible device comprises an actively controllable section.
 32. The medical system of claim 31, wherein the actively controllable section comprises an actively controllable stiffness.
 33. The medical system of claim 24, wherein the instructions for determining the deformation force comprise instructions for determining a stiffness of the section of the flexible device, the section of the flexible device comprising an actively controllable stiffness.
 34. The medical system of claim 24, wherein the section of the flexible device comprises an actively controllable stiffness, and wherein the flexible device includes: a second section comprising a second stiffness; and a third section comprising a third stiffness, the third stiffness being greater than the second stiffness, and the actively controllable stiffness being greater than the third stiffness.
 35. The medical system of claim 24, wherein the computer executable instructions further comprise instructions for determining the shape of the flexible device based on information from a shape sensor of the flexible device.
 36. The medical system of claim 24, wherein the computer executable instructions further comprise instructions for generating a composite model of the patient anatomy based on the registering, and wherein the composite model includes the model of the flexible device positioned within the model of the candidate anatomical passageway.
 37. The medical system of claim 24, wherein the computer executable instructions further comprise instructions for determining a restorative force corresponding to the deformation force exerted by the section of the flexible device, and wherein the instructions for registering the model of the candidate anatomical passageway of the plurality of anatomical passageways to the model of the flexible device is further based on the restorative force.
 38. A medical system comprising: a flexible device configured to be positioned at least partially within an anatomical passageway of a plurality of anatomical passageways within a patient anatomy; and a memory device including computer executable instructions, the computer executable instructions for performing operations comprising: determining a deformation force exerted by an actively controllable section of the flexible device; and registering a model of a candidate anatomical passageway of the plurality of anatomical passageways to a model of the flexible device based on: a shape of the flexible device; and the deformation force exerted by the actively controllable section of the flexible device.
 39. The medical system of claim 38, wherein the computer executable instructions further comprise instructions for identifying the candidate anatomical passageway in a model of the patient anatomy by matching a plurality of points of the model of the candidate anatomical passageway to a plurality of points of the model of the flexible device.
 40. The medical system of claim 39, wherein the model of the patient anatomy comprises an undeformed model of the patient anatomy including a model of the plurality of anatomical passageways.
 41. The medical system of claim 39, wherein identifying the candidate anatomical passageway comprises comparing the shape of the flexible device to a shape of a model of each anatomical passageway of the plurality of anatomical passageways.
 42. The medical system of claim 38, wherein the computer executable instructions further comprise instructions for determining a restorative force corresponding to the deformation force exerted by the actively controllable section of the flexible device, wherein the registering the model of the candidate anatomical passageway of the plurality of anatomical passageways to the model of the flexible device is further based on the restorative force.
 43. The medical system of claim 38, wherein the actively controllable section is positioned at a distal portion of the flexible device. 